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[57] ABSTRACT 

A computer operable method for integrating and automating 
test procedures within a computer application program. 
Instantiated test operation objects of an object class defined 
by the present invention correspond to functions to be tested 
within the computer application program. The test operation 
objects are instantiated by calls to functions in a test opera- 
tion runtime library (DLL). The test operation objects 
include interface method functions which execute the 
intended test operation and simulate any required data, file, 
or memory I/O. Other aspects of the methods of the present 
invention provide rules which permit decisions as to the 
applicability of each test operation given the state of the 
application program or the context of the test operation 
sequence. The various test operation objects are selected to 
perform a sequence of test steps. In one mode of operation, 
the methods of the present invention randomly select among 
all the instantiated test operation objects. In another mode of 
operation, the methods of the present invention "playback*' 
a previously recorded sequence of test operation objects to 
permit reproduction of failures in previous test sequences. In 
a third mode of operation, the methods of the present 
invention permit a user to modify or create "playback" files 
to customize a test case for a particular focus. 

29 Claims, 12 Drawing Sheets 
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METHOD FOR INTEGRATING AUTOMATED 
SOFTWARE TESTING WITH SOFTWARE 
DEVELX)PMENT 

HELD OF THE INVENTION 

The present invention relates lo the field of computer 
software development and testing, and in particular, to a 
method for integrating automated software testing into a 
product as it is being developed to thereby simplify subse- 
quent software product testing. 

PROBLEM 

It has been common in the computer software arts that 
software product development was a distinct and separate 
process from software product testing. In the standard soft- 
ware product development lifecycle, development engineers 
would iteratively develop and refine a software product 
according to requirement or functional specifications. The 
development engineers frequently would test the product 
under development either in an ad-hoc manner or in a more 
rigorous manner. In either case, when the product was 
deemed to be completed by the development group of 
engineers, the software product was turned over to the 
testing process, usually a separate set of engineers from the 
group that developed the software product. Most, if not all, 
of the testing process utilized in the development phase 
would be discarded and the test engineers would begin anew 
evaluating the software product as against its corresponding 
product specifications. 

In a typical test environment, the test engineers are 
provided with fittle or no internal information regarding the 
structure and design of the code comprising the software 
product. The testing effort would then proceed in a "black 
box" manner by applying test data (an external stimulus) and 
observing the output of the software product for proper 
response. In its crudest form, a test engineer determines 
("dreams up") potentially interesting test inputs and applies 
them to the software product while observing the output 
behavior of the software product for proper operation. This 
form of manual testing permits wide flexibility for the test 
engineer in creating input stimuli but provides little assis- 
tance to the test engineer in reproducing problems found 
during the test sequence. Manual test requires the test 
engineer to carefully note the sequence of test inputs which 
led to a specific problem test result. 

Test engineers have developed or utilized a number of 
tools to assist in such "black box" testing. To automate the 
above discussed manual test process, scripting and macro 
languages are frequently used. The script/macro tool allows 
some degree of automation in the test sequence to aid in 
reproducing intermittent failures. A macro tool permits some 
added flexibility in the use of variables to customize the 
operation of the script based on externally supplied variable 
inputs. Some software products (e.g. word processing pro- 
grams or spreadsheet programs) offer built-in macro features 
to reproduce sequences of user input keystrokes. Other 
program user interface (Ul) environments (such as the 
Xwindows programming environment) provide for redirec- 
tion of a program's input from a script/macro file and thus 
may automate the testing of specific user keystroke 
sequences. 

Simple "BAT' files in the MS-DOS® environment or 
"shell scripts" in the UNIX® environment are exemplary of 
such scripting tools used to partially automate the software 
product testing process. Macro facilities in the "BAT" file or 
"shell script" interpreters and other macro replacement 
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programs such as "m4" or "perl" in the UNIX® environment 
are exemplary of macro facilities used to enhance the 
scripting facilities in automating software product testing. 
However, such scripting/macro based testing tools still 

5 provide little or no assistance to the test engineer in observ- 
ing the software product output to automatically detect 
success and failure of each test. Collection and analysis of 
the test results remains largely a manual procedure. Corre- 
lation of the gathered test output with the timing and 

10 operation of the software product is difficult if not impos- 
sible. 

An additional problem with scripting/macro tools arises in 
that the tool itself as well as the test scripts themselves must 
be ported to each computing environment in which the 
software product is to be tested. For example, a particular 
operation in a PC based Microsoft Windows® application 
may be tested by simulating an "OPEN" menu operation 
while a similar function may require an "OK" menu opera- 
tion in an Apple Macintosh® computing environment. The 
test scripts would have to change to test the same function 
in these two exemplary computing environments. In other 
words, the testing of underlying functions of an application 
may have to change as the user interface changes over a 
variety of computing environments. This creates an addi- 
tional burden in porting and maintaining the test tools and 
test scripts along with the software products. In addition, the 
porting and maintenance of the scripting/macro tools and 
test cases can be a source of additional errors which may be 
erroneously attributed to the failure of the software product 
under test. 

Scripting/macro based test tools also tend to vary depend- 
ing upon the external stimuli needs of each software product 
under test. The user interface for the test engineer using 

2j "black box" automated test tools tends to vary as the needs 
of each software product vary. Test engineers therefore must 
potentially leara a new user interface mode of operation for 
each software product to be tested. 

All such "black box" testing methods, with or without the 

40 aid of automated scripting/macro testing tools, are typically 
performed without knowledge of the software product's 
internal code structure. This lack of structural knowledge 
can make testing more cumbersome and time consuming. 
The lack of structural knowledge precludes certain styles of 

45 testing which may focus on particular error prone aspects of 
the software product revealed only by knowledge of the 
software product's internal structure. Sometimes, for 
example, an error in one operation of the software product 
may not be externally observable in the output of the 

50 software product until subsequent operations are performed. 
The combination of the operations may eventually reveal the 
problem in an external manifestation, but the sequence of 
event may be lengthy back to the actual cause of the 
problem. Thus the use of external stimuli ("black box") test 

55 methods, even in association with various automated test 
tools, does not permit the test engineer lo exercise judge- 
ment with respect to problems more easily located through 
testing with knowledge of the software product's internal 
code structure. 

60 In addition, as software products evolve with ever increas- 
ing complexity, the burden of testing is being shifted more 
toward the development teams. It is simply impossible with 
"black box" testing techniques to exhaustively test all pos- 
sible inputs (external stimuli) of many software products 

65 regardless of the size of the testing staff. Even testing of a 
large portion of the possible inputs is a difficult task in many 
cases. Therefore, the software development/testing lifecycle 
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has begun to shift more of the testing efforts onto the operation object (of a predefined test API object class) 
responsibility of the development staff. For example, it is corresponding to each operation to be tested. The test 
more frequent now that "acceptance testing*' (testing for operation object identifies an execute interface method 
fundamental functionality) is made a part of the responsi- (provided by the application programmer) to be invoked to 
bility of the development staff. An added benefit of shifting 5 execute the desired operation. Other interface methods of the 
some test burden to the development group is realized in the test operation object identify functions which are to be 
knowledge of internal code structure of the software prod- performed to simulate the generation of input data and the 
ucl. The code structure knowledge of the development group acceptance of output data in response to operation of the 
may be applied to focus the testing effort on test sequences execute interface method. For example, one interface 
and scenarios most likely to create problems. In addition, the lO ^^^^ provides functions to be invoked for the manual or 
developers may construct the test cases to detect the failure automatic generation of input data to test the operation, 
in the test sequence as early as possible. It is a problem for Another interface method of the test operation object pro- 
developers to create a standardized lest interface for the ^^^^^ functions to be invoked in response to I/O requests of 
automated performance of test sequences in a manner that t^e operation under test. Another interface method of the lest 
permits easy reproduction of the failed test sequences. 15 operation object defines a set of rules to permit decisions to 

. . f *u u J- • *u . J • . t>e made in the invocation of the test object. 

It IS apparent from the above discussion that a need exists _ ... ■, - 

for an automated testing tool which permits a standardized application program is designed by the development 

interface for the generation and execution of test cases, engmeers with the test operation objects defined as part of 

which permits random sequences of testing, which permits application program. The application program is then 

easy reproduction of failed test sequences, and which auto- 20 compiled and linked with the test tool library (DLL) to 

maticaUy senses the success or failure of each test step. P^^V"^^ '^^"^^'"^ object class naanipulation test methods 

of the present invention. By defining the test operation 

SOLUTION objects as discussed above, the application programmer has 

provided information (in the form of a collection of instari- 

The present invention solves the above identified prob- ^5 tiated objects) useful to the test methods of the present 

lems and others to thereby advance the state of the useful invention to invoke each test operation and detect success or 

arts by providing an easy to use, coding standard and user failure of each tested operation. Since the test operations are 

interface for a software developer to integrate testing of a designed into the apphcation program, and the library func- 

software product into the development of the product. The tions which implement the test operation classes of the 

testing application program interface (test API) is used by an 3^ present invention are available in a variety of computing 

application program developer to integrate a standardized environments, the application program may be more easily 

test interface into the software product in its development tested in many environments without the need to port unique 

phase. Use of the test API by the application program test tools to each new computing environment. The test tools 

invokes standard functions in a dynamic linked library can be readily available in any computing environment to 

(DLL) to initiate a standard test user interface when the 35 which the development staff chooses to port the application 

application program is run. The test tools are integrated into program. In addition, the test tools of the present invention, 

the application program and are therefore ported to any new unlike scripting and macro methods of the past, do not 

computing environment by simply porting the application require the development or testing teams to port the test 

program to that environment. The development engineer, cases between the various computing environments on 

possibly in cooperation with the test engineers, thereby which the program is to be tested. The test tools of the 

designs the test capability into the application program from present invention are available for all platforms on which the 

its inception. These test hooks, combined with the standard, application program may be run. The test suites are recorded 

easy to use, graphical user interface of the test tools of the in a portable manner by the test tools of the present invention 

present invention, permit any engineer in the product devel- in so-called playback files. 

opment team to test the product. Development engineers ^5 The application program is designed by the application 
may more effectively test the application program during its programmer to invoke the test tool at an appropriate stage in 
development phase to reduce the number of problems dis- the runtime startup or processing of the application program, 
covered later in the product's lifecycle. Test engineers may ' -phe test tool may then be operated in one of three modes: 
apply the same easy to use graphical interface to more namely, random automated test operation selection, play- 
thoroughly test the functionality of the product. t?ack automated test operation selection, or manual playback 

The test API functions in the DLL provide a consistent, creation mode, 
friendly, easy to use interface for testing of the application in the random automated test selection mode, the test 
program. The DLL test functions may be operated in a methods of the present invention pseudo randomly select 
random mode to generate pseudo-random test sequences. from among the set of instantiated test operation objects 
The DLL test functions automate recording of the pseudo- 55 defined by the application programmer. As each test opera- 
random seed and the corresponding sequence of test steps tion object is randomly selected, the execute function 
for easier reproduction of problems generated in the random method associated with the object is invoked to perform the 
sequence. The DLL functions can be used to automatically developer's designed test function. The random selections 
determine the success or failure of the corresponding test are logged in a playback file to permit later reproduction of 
step by sensing a return value from the invocation of a a failed sequence. In addition, the results of each execute 
corresponding sequence of execution in the application function method invocation are logged in a file by the 
program. In addition, the appUcation program may directly methods of the present invention so that a sequence may be 
invokeother DLL functions to record an application failure. reproduced if it produces a problem in the application 
This alternative method is referred to herein as the assertion program under test. This process continues randomly select- 
method. 65 ing test operation objects for which the corresponding 

The application programmer determines the operations to execute function is invoked until directed to stop by a user 

be tested in the application program and instantiates a test or other termination conditions are met. 
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In the playback automated lest operation, the lest opera- knowledge of the internal structure of the application pro- 
lion selections made in a previous test sequence are replayed gram. By contrast, it is common in present software devel- 
from ihe playback file created during an earlier performance opment and test environments that the quality assurance or 
of the test sequence. This mode of operation is used, for testing team operates using "black box" techniques 
example, lo permit reproduction of test sequences which led 5 (applying stimuli and observing results without such knowl- 
to erroneous results in previous invocations. edge of the structure of the application program). In this 

The manual playback creation mode of operation permits typical software development and lest environment, the 

a user of the present invention to create or edit a playback development team helps determine the requirements and 

file. This mode allows the user finer control over the test functional specification of the application program as 

sequencing lo more rapidly debug an application program. depicted in element 100. In element 102, the development 

The methods of the present invention enable automation develops appropriate programs and data structures lo 

of the testing of an application program while integrating the i^^plement the requirements and functional specifications. In 

test cases with the program development. This integration of conjunction with the development thereof, the development 

the test case and program development permits more thor- ^^"^ or vaUdates the program against the require- 

ough analysis and regression of the application program. "^^"^^ functional specifications. In this typical applica- 

The automation aspects of the test methods of the present P^°g^^"^ development process, the development team 

invention permit easier, unattended test operation to more completed its task and provides the application 

effectively test an application program. program to the quality assurance or testing team for inde- 
pendent testing and verification of the application program. 

BRIEF DESCRIPTION OF THE DRAWINGS 20 In element 104, the quality assurance or testing team 

performs a regression test lo verify that previously reported 

FIG. 1 is a flowchart depicting the lifecycle flow of an and fixed problems have not recurred in this new version of 

application program through development and test as pres- the application program. Next, at element 106, the quality 

ently known in the art; assurance or testing team performs new testing of the 

FIG. 2 is a flowchart depicting the lifecycle flow of an 25 application program to reveal potential new problems aris- 

apphcation program through development and test as ing in the current version of the application program. Due to 

improved by the tools and methods of the present invention; the nature of the prior testing tools, the test team must 

RG, 3 is a block diagram depicting the relationship initiate measures at element 108 (often manual) to document 

between script oriented external test tools and an application ^^^P^ required to reproduce the new problems discov- 
program as known in the art; 

FIG. 4 is a block diagram depicting the relationship ^ element 110. the quality assurance or testing team 

between script oriented external test tools and an application determines whether or not the number and seventy of 

program as improved by the tools and methodsof the present P'°'''*''°' ^'^ ^'^^ '° P*™" '^'^^^ °^ '^e 

invention- apphcaUon program to potential users. If it is determmed 

„ _ ' . „ , „ „ 35 that the number and severity of problems revealed is not too 

HG. 5 IS a flowchart depicting the control flow of an ^igh, the quality assurance or testing team will release the 

application pro-am with test operation objects embedded appUcation program to potential users as depicted in element 

therein in accord with the present mvention; ^4 number or severity of the problems revealed is too 

FIG. 6A IS a flowchart showing additional detail of the high, the quality assurance or testing team does not release 

random test operation element of FIG. 5; 40 the application program lo potential users, but rather reports 

FIG. 6B is a flowchart showing additional detail of the the open problems to the development team as depicted in 

random test operation of FIG. 5; element 112. In parallel with release of the application 

FIG. 7 is a flowchart showing additional detail of the program as depicted in element 114, the quality assurance or 

playback test operation of FIG. 5; testing team will also report the open problems to the 

FIG. 8 IS a flowchart showing additional detail of the developers as shown by element 112. With the report of open 

manual test operation of HG. 5; problems returned to the development team, the develop- 

n . J. , L • . ^e^f" repeats its previous efforts in element 102 to 

■ . f ^^.^^'^^^.f^P^^y showing an exemplary user ,,.^,3^ j^e application program to resolve problems 

interface which permits the user to define group and test reported by the quality assurance or testing team. This cycle, 

operation weights for the random test operations of HG. 5; 5^ elements 102-U2, repeats untU the quality assuranci; or 

RG. 10 IS a screen display showing an exemplary user testing team finds no remaining problems in the application 

interface which permits the user to define options for the program worthy of reporting to the development team, 

random test operations of FIG. 5; and FIG. 3 depicts a typical computing environment 308 in 

FIG. 11 is a screen display showing an exemplary user which the quality assurance or testing team performs their 

interface which permits the user to customize a playback test 55 testing function. A typical testing approach involves the use 

sequence for the playback test operations of FIG. 5. of a script oriented test tool 304 to apply external stimuh lo 

application program 306. The application program receives 

DETAILED DESCRIPTION OF THE the externally applied stimuli as a simulated user input and 

INVENTION generates an appropriate response by performing its desig- 

OVERVIEW—PRIOR ART 60 nated function on that provided input. The script oriented 

FIG. 1 describes the overall flow of a typical software test tool 304 detects the response generated by the applica- 

development and test life cycle for an application program tion program 306 to verify proper operation of the applica- 

product. A vertical dashed line in the middle of FIG. 1 tion program 306 in response to the applied stimuli. Script 

delineates tasks performed by a development team on the oriented test tool 304 may receive inputs from test suite or 

left side of the dashed line, and tasks performed by a quality 65 script files 300, which describe a sequence of test steps to be 

assurance or a testing team on the right side of the dashed executed. The test tool 304 also logs the test results and 

line. It is axiomatic that the development team has extensive progress into log files 302 for further analysis. 
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Scripl oriented lest tool 304, typical of current test the application program during development. By so embed- 

procedures, is external to application program 306. Due to ding test operations within the application program, the 

this limitation, script oriented test tool 304 can only apply development team (with the aid and consultation of the test 

external stimuli that are expected, and accepted, as external team) determines appropriate functions and operations to be 

user inputs by the application user program 306. No inter- 5 tested. In addition, the development team determines the 

mediate levels of functionality testing below that which is expected proper response for each identified test operation 

made visible to the user interface is possible with this embedded within the application program .yThe testing tools 

external testing approach. Similarly, script oriented test tool and methods of the present invention further provide a 

304 can only detect responses from application program 306 standardized, easy to use, graphical, user interface for the 

that are made externally visible by the application program, lo performance of test operations on the applications program. 

Script oriented lest tool 304 is thus limited in its internal By embedding test operations within the application 

knowledge of the structure and design of the application program, complete automation of test procedures can be 

program 306 and limited to only those stimuli and responses achieved both in provision of stimuli by calling functions, 

that are provided to the external user interface of application and in the capture of test results by verifying the return code 

program 306. For example, a typical response of an appli- 15 value of a called test operation function or by direct assertion 

cation program 306 may be no more than the update of a function calls of the application program to detect failures, 

display screen to a user interface. Such a response may be These automated test procedures may be used throughout 

detected by script oriented test tool 304 only by capturing the development portion of the software life cycle by the 

the screen dump and storing it for subsequent manual review development team or by the test team. This integration of the 

by a test operator. In this manner, automation of the testing 20 testing of the application program with its development 

procedures is made more difiBcult because the detection of eliminates the distinct division of labor typical of prior 

the application program 306 response is frequently diflScult product testing approaches. Knowledge of the program 

to automate. structure is no longer required to effectively and robustly test 

Hand generated testing is a variant of the test tool com- the application program. The development lean] and the test 

puting environment depicted in FIG. 3. In a hand generated 25 team may act in concert as a unified group to assure quality 

lest environment, a test engineer hand generates a desired in the application program product, 

external stimulus rather than receiving a stimulus from a FIG. 2 depicts a software development and test process 

pre-defined script file 300. Such hand generated testing which may utilize the tools and methods of the present 

limits the reproducability and automation of the test proce- invention. Unlike the process depicted in FIG. 1, there is no 

dures in terms of the generation of external stimuli. The test 30 vertical dashed line indicating the separate responsibilities 

engineer must be certain that all steps which lead up to of the development team and those of the quality assurance 

production of a problem are noted so that the problem may or testing team. Since there is no longer the need for strict 

be reproduced. So called "macro" languages are yet another division of responsibilities under the methods of the present 

variant of the test procedure depicted in FIG. 3. Macro invention, the two teams are referred to under the methods 

language lest tools are similar to the script oriented test tool 35 of the present invention as the product development group. 

304 but with reduced capabilities for detecting and recording The product development group, by operation of elements 

the response from application program 306. In essence, 200 and 202, defines the requirement specifications for the 

macro language test tools provide a shorthand notation for application program and develops appropriate program and 

defining a test script of external stimuli to be applied in data structures to implement those requirements. In element 

sequence, 40 202, the product development group embeds test operation 

The well known apphcation program development test object definitions within the program structure of the appli- 
cycle and associated tools depicted in FIGS. 1 and 3, cation program under development. These test operations 
therefore, have several weaknesses as compared to the object definitions are derived from object class definitions 
techniques disclosed by the present invention. Specifically, provided by the tools and methods of the present invention, 
script oriented test tool 304 is only capable of testing those 45 By embedding the test operation object definitions within 
functions made externally visible by the application program the application program (in element 202), the product devel- 
306. Similariy, detection of responses generated by appli- opment group aids in the testing of the application program 
cation program 306 is limited to those responses made by providing interfaces for testing functionality not other- 
visible by the application program. Some such responses wise visible external to the application program structure, 
may be difficult to capture or detect in any automated way 50 TKis embedding of test operation objects allows automation 
by script oriented test tool 304. An additional problem with of the testing procedure for generating test stimuli as well as 
the script oriented test tool 304, of FIG. 3, arises in the need automated capture and analysis of the test results. In 
. to port the test tool 304 to each new computing environment addition, the embedding of the test object definitions permits 
308. The application program 306 development team nor- a broader range of test functionality to be provided as 
mally ports the application program 306 to several different 55 compared to the external testing approach depicted in FIGS, 
computing environments 308. In addition, either the quality 1 and 3. By operation of elements 204 and 206, the product 
assurance or testing team or the development team must port development group performs regression testing of the appli- 
the script oriented test tool 304 and/or the script files 300 to cation program by use of the test tools and methods of the 
the same collection of computing environments 308 inde- present invention as embedded within the application pro- 
pendent of the efforts to port the application program 306. 60 gram by the test operation object definitions. In elements 
This imposes an additional work load upon the product 204 and 206, the product development group determines 
development and test in addition to the normal load for whether too many problems are revealed through a verifi- 
developing the product and identifying problems in the cation and regression testing. If too many problems are so 
application program 306. revealed, the product development group repeats the steps of 
OVERVIEW—PRESENT INVENTION 65 elements 202-206. The product development group next 

The testing tools and methods of the present invention tests the application in elements 208 and 210 to determine if 

involve embedding of test operations within the structure of too many new problems have arisen in the current version of 
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the application program. If the application program is deter- the testing tools of_the_p2e sent invention . Actable is builf 

mined to be sufficiently problem free by operation of ele- withiiTtHeTest tools of the present invention Jn which ea2h 

ment 210, then the program is released to application users eriti^'^*5ff^p^?iSs^^^(^^^ operation 

as duplicated in element 214. In either case, all newly o^j^lK'fe discussed'-below-in" further' detail, grouping/'of 

discovered problems are reported for the development group s rejatcdj)bj ects.and„wd ghtm 

to consider as shown in element 212. The reported open /utilizedTn the automated testing procedures to selectjppro- 

problems returned to the product development group to priat^'1^stVn^5'*6'^ir^^ 

repeat the program development cycle in hopes of repairing '^Element-504-determ'ines whether the user has requested 
the problems. that testing begin immediately or be deferred until some 
The tools and methods of the present invention permit the lO subsequent time in the processing of the application pro- 
shift in testing paradigm depicted as between FIGS. 1 and 2. gram. If the testing is to begin immediately, processing 
Whereas under prior approaches all testing, regression continues below with element 510. If in the alternative, 
testing, as well as new testing, was the responsibility^of^ testing is deferred until some subsequent time, processing 
distinctjualuy_assurance_a^^ topis and continues with elements 506 and 508 repetitively until it is 
methlods^ofjthe presenr iny|^iitipn^^^^^^^ 15 determined by element 508 that the user has requested the 
testing_to ^be_ performed at commencement of testing procedures. Until that time, de- 
program development lifecycle,/ ment 506 represents all normal processing of the application 
"""The'toolsand^Siethods of the present invention are shown program performing its intended function. The determina- 
in a block diagram of FIG. 4. Application program 404 is tion of whether to begin testing immediately or to defer until 
operable within computing environment 408. Embedded 20 a later, user specified, time is a matter of design choice. One 
wiUiin_appJ ication-p rogra m-404-arer^est;^pe ratiooTobjects of ordinary skill in the art will recognize the potential value 
406 adapj^J^^^ auto mated testin g of functions with^ in deferring the start of testing until a particular state of the 
applicati^progTam^404. The test operation objects 406 are application program is created (by manual operation of the 
de7ived"(by the application programmer developers) from an program). Many equivalent design choices may be made by 
object class provided in a programming library of the 25 those practicing the present invention to implement a desired 
present invention. The library is provided in the form of a delay in the commencement of a test sequence, 
dynamic link library (DLL) to be invoked by application When testing procedures are begun, elements 510 and 514 
program 404. The lest operations object library 406 receives are operable to determine which of three possible modes of 
pre-recorded.ps^udo-random test suite files or playback files testing operation are to be performed. The test tools of the 
400, and logs^ all testing procedures to playback files and 30 present invention may be operated in a random auto-test 
results to^^'g files 402? As shown in FIG. 4, the tools and mode, a playback auto-test mode, or a manual playback 
methods of the present invention embed the test operations creation mode. Element 510 is operable to determine if 
406 within the application program 404 at the time of its random auto-test mode has been selected. If so, element 512 
development by the product development group. By embed- is next operable to perform the random auto-test procedures, 
ding the test operation objects library (DLL) 406 within 35 If not, then element 514 is operable to determine if the 
application program for 404, the developers may expose playback auto-test mode has been selected. If so, element 
more functionality for testing than is otherwise possible in 516 is operable to perform playback auto-test operations, 
prior test methods which exercise only externally visible Otherwise, element 518 is operable to perform manual 
user interface functions. In addition, the results of each test playback creation procedures. Once begun, elements 512, 
operation may be automatically verified by validation of the 40 516, and 518 are operable until completion of the testing 
return code value from the invocation of each lest operation procedures. 

object. An additional benefit of the test tools and methods of FIG. 6A depicts a flowchart of the detailed operation of 

the present invention is derived from the fact that porting of random auto-test element 512 of FIG. 5. In the random 

the application program to a particular computing environ- auto- test mode, test operation objects are randomly selected 

ment 408 automatically performs the required porting of the 45 for invocation to test the application program. The user may 

test operations embedded within the application program. provide weighting values to increase^the Jrequency-of-cer- 

There is no additional effort required to port testing tools to tain groups of tests l^eing'selecled as ^omp_aredjp_other_y 

each computing environment along with the application groups ofTests^^as-will-be-discU^ed'^m^aSditional detail 

program itself. 'below.-Element-600 is operable to prompt the user for input 

TEST TOOL OPERATIONS 50 used to adjust the default_.weighting-values-appliedJo_each 

FIG. 5 is a flowchart of the operation of an application test o peratio n^jegt^B^se^ on thejdefaiUtAveigh ting values, 

program which has embedded the test tools and methods of plus"any adjustnients entered by the iiseTm operation of 

the present invention. Element 500 represents the normal c]pm^^j^^^^^M '6(^2 is hext" oprable to generate 

initialization procedures for the application program. These ^igVted jejectionla^^ for randoml y sej ecting~eacl? 

operations include whatever data and code initialization is 55 of the test ^^^r^tipiii^bje^^^ 

required to properly perform the intended function of the 'provided 'by_;jh(0^^ — --^ 

application program. Element 502 is next operable to instan- A weighting table specifies a range of random number 

tiate all the test operation objects defined by the product values which correspond to each entry in the table. The 

development group (and derived from the object class ranges of possible random numbers is equal to the total of 

definition provided by the DLL). The instantiation of the test 60 the weight values for all entries in the corresponding table, 

operation objects involves creating the objects and the A weighting table is first generated for the current group 

associated interface functions and making them known to weight values to determine which group will be selected, 

the test tools which actually perform the testing sequences. When selecting a test operation object, a first random 

I n additioj L-as_d_iscussed below,j&lemenJ502-is-operable to number is generated in the range of zero through the total of 

(j gfi^W g^PS of related gbjects^^PsTmplifying-the-user 65 all group weights (minus one). The group whose assigned 

interface tO~the-test-tool.^nstantiating each test operation range of numbers encompass the randomly generated num- 

object provides the definition of the objects to be tested by ber is then selected. After the group is randomly selected, a 
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weighted selection table is generated for the test operation 
objects according to the current weight values for test 
operation objects in the selected group. A second random 
number is then generated to select from the test operation 
objects defined as members of the selected group. The test 
operation object whose assigned range of numbers encom- 
pass the second randomly generated number is then selected. 
For example, if test objects "A" and "B" are in group "X" 
and test objects "C and "D" are in group "Y", and the object 
and group weights are as follows: 



Object/Group 



Group Weight 



Object Weight 



60 
30 



20 
10 
100 
150 



Then the group weighted selection table would reflect the 
group weights as follows: 



Group 



Random Number Range 



0-59 
60-^9 



The test operations weighted selection table if group X were 
randomly selected would appear as follows: 



Group X Object 



Random Number Range 



0-19 
20-29 



The test operations weighted selection table if group Y were 
randomly selected would appear as follows: 



Group X Object 



Random Number Range 



0-99 
100-249 



Element 604 is next to operable to prompt the user to enter 
a random seed value. The random selection in the random 
auto -test mode is performed via pseudo-random number 
generation techniques. A sequence of "random" numbers 
generated by such a pseudo-random number generator tech- 
nique may be repeated if the random number generator is 
provided with the same seed value. Such techniques are well 
known to those of ordinary skill in the art and need not be 
discussed further. In order to assure reproducability of any 
problems found, the random number seed value used to 
previously produce the problem may be manually reentered 
by the user so as to reproduce an identical "random" 
sequence of test operations. Element 606 is next operable to 
prompt the user for input parameter values indicating the 
threshold for the number of errors allowed and the total 
count for the number of test operations to be performed. The 
threshold and count values are used to determine the length 
of time the test procedure will be run and a threshold limit 
at which testing will be halted. If the number of problems 



12 



revealed is below the threshold value testing will proceed, 
otherwise testing is halted when the number of errors 
exceeds the threshold value. Similarly, the total test count 
value is used to determine the number of random test 
operation objects to be invoked in the random auto-test 
mode. 

Elements 608-626 are performed repetitively until testing 
procedures are halted. Element 608 is operable to determine 
whether the specified error threshold has been exceeded. If 
10 so, testing operations are halted and the test function has 
completed. If not, element 610 is next operable to determine 
whether the total count of test operations has been 
exhausted. If so, testing is completed. If not, processing 
continues by selecting another test operation object invoca- 
15 tion. 

For each test operation object invocation, elements 
612-626 perform the actual test operation. Element 612 is 
first operable to adjust the weighted selection table for any 
changes in the weighting values. Changes in the weighting 
20 values are the means by which rules, discussed below in 
additional detail, may alter the testing strategy as the test 
process proceeds. Element 614 is next operable to generate 
a pseudo-random number value for use in the weighted 
selection table to select the next test operation object. The 
25 pseudo-random number value may be used as an index value 
into the weighted selection tables as described above, or 
otherwise utilized with data structures, well known by those 
of ordinarily skill in the art, in a manner indicative of the 
weighted selections. Element 616 is next operable to record 
30 the selected test operation object in a playback file. A 
playback file may be utilized in a subsequent test procedure 
to reproduce a problem revealed in random test sequencing. 
The selected test operation object is recorded in the playback 
files before the test is performed to assure that the playback 
35 file records each test in sequence before the test operation 
execution potentially hangs the application program 
(indicative of a problem in the application program). The 
test procedures and methods of the present invention also 
permit system exception handling to be intercepted by the 
40 test methods to detect unexpected results in the test opera- 
tion object invocation. This capability of the present inven- 
tion depends upon the computing environment capabilities 
on which the application program (with the test methods 

e m bedded ) is run. 

45 Element ^^^ jg^is .next^Qperable to invoke the^xecute 
interface function of t he j§g((^^jj ff t^tg^atio^^ 

P^^l^y^^^^^^JS^J^'^*^^"'"^^ defined by the .product 
de^jQpment group correspondin g to this selected test opera - 
tion-obiect?The„exe cute f anction.is„defined,by the product 
50 developm*ent*pioup'to^erform'wh^^^ process- 
ing^is^^^nSpriale td^B^ 

to tH^ tel^^lfi^^ of an execute function 

are disciT^d below with respect to exemplary test operation 
objects. The result's of the execution interface function is 

55 recorded by operation of element 620 in a resultjpg^je^. An 
execute interface function may indicate its success^oKfailiire^ 
by returning a specific result code as the function value or by 
calling an assertion function in the library methods of the 
present invention to indicate an erroneous test result. The 

60 result log file may be inspected later to determine whether 
the test sequence failed or succeeded, and in addition, may 
be used to determine at which step in the test procedures, as 
indicated in the playback file, the test sequence failed. 
Element 622 is operable to determine whether the test 

65 execute function result represents a successful or failed 
invocation of the execute interface function. If the execute 
interface function invocation resulted in an error, element 
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624 is next operable to increment the error counter. If the Test operation objects are objects derived from a standard 
execute interface function resulted in a successful C++ class definition provided by the library (DLL). The 
completion, then the error counter is not incremented and object class may be used by the product development group 
operation of element 624 is skipped. Element 626 is finally to derive specific definitions and instantiations of test opera- 
operable to decrement the total test count. The test process 5 tion objects. The test tools of the present invention define a 
then continues by looping back to elements 608 and 610 minimum of four interface function methods for use by the 
which verify that test processing should continue or should testing tools in performing the required test operation. The 
be halted based on the error count and the total test count product development group members define their own test 
remaining. operation object class which implements at a minimum the 

FIG. 7 depicts a flowchart providing additional details of lo execute interface methods described above. The other three 
the operation of elements 516 of FIG. 5, which performs of the four basic interface methods are provided with default 
playback auto-test mode processing. Element 700 of FIG. 7 definitions in the object class. The programmer may use the 
is first operable to prompt the user to e nter the^ namc of a standard object definitions of the three other interface 
previously recorded playback file. The ^^|^K^Me con- methods, or may replace them (override them) as appropri- 
tains information used by the methods of'the-presenl inven- is ate for the particular objects test goals. The specific func- 
tion to reproduce a previously recorded test sequence. The tionalily within each of the four basic interface functions is 
playback auto-test mode is most typically used to reproduce provided by the product development group engineer to 
a problem previously produced in a sequence of test steps actually perform the requirgcLJLcst_p peration and return a 
(such as the pseudo- random auto-test mode or manual succe_ssj)r_failure-result.-pieAbasietfeui^ 
playback creation mode discussed above) and recorded in a 20 aJrerex^ution,^ata_input,.file-and-memory-I^^ 
playback file. Test sequence reproducabilily is key lo the fhe execute interface function must be provided in the 
product developmerU.group m locating and repairing pro- derived class object. There is no default implementation of 
grammmg errors (bug^'causmg the reported problem. an execute function because the execute function contains 

Elements 704-710 -^re next operable repetitively until all of the code necessary to perform a particular test opera- 

testmg procedures have been completed. Element 704 is 25 tion. The execute interface function performs only the 

operable lo determine whether additional test operation precise function calls required to execute the desired test 

objects remam to be processed from the playback file. Each operation function. Any input or output requirements are 

entry m the playback file represents the invocation of a handled through the data input or file and memory 1/0 

selected test operation object. Element 704 then determines interface functions. The following is a simple example of an 

whether additional entnes are yel lo be processed in a 30 execute function used to perform an open file operation, 
selected playback file. If no further test operation objects 
remain unprocessed, processing of the playback auto -test 
mode is completed and testing halts. If additional entries 



remain in the cunrenlly selected playback file, element 706 dword copcnFiic::Exccuic( ) 

is next operable to invoke the execute interface function to 35 ^ „ , n . .... ... 

f , 1 . , . . // l^g will wntc the provided string to the logftlc and return FALSE 

perform the next selected test operation. if the operation should skip execution. 

Element 708 is operable to determine whether the result if (i^g(m_stiOperationName)) 

of the previous execute interface function indicated success { 

or failure. If the execute interface function returned a . GetAppiication()->OpenFUe(in_strFiieName); 

successful completion code, processing continues by loop- 40 return ErrorC ); // Return success or failure. 

ing back to element 704 to thereby continue the playback } 

auto -test mode by selecting the next test operation from the ' — 

playback file. Conversely, if the execute interface function 

invocation indicates an erroneous result, element 710 is next exemplary execute function (the Execute function of 

operable to record the erroneous result in a results log file. 45 COpenFile test operation object) simulates a users 
Processing then continues by looping back to element 704 lo selection of a file/open menu item exactly as though the user 
complete the playback auto-test mode processing of the selected the file/open menu item by operation of the 

selected playback file. application program. 

FIG. 8 is a flowchart providing additional detail of the At least two functions may be defined as part of the data 

operation of manual playback creation mode element 518 of so input interface method. These functions are invoked indi- 
FIG. 5. Element 800 of FIG. 8 is first operable lo prompt the ^^ctly by the invocation of the execu te interface fun ction 
user to select a particular pre-existing playback file (for wjien datajnpmjsj^qy^^ data^ng^ut 
editing) or to create a new playback file. Next, element 802 meth"c>d~c'an autornat|canyg3^ 

prompts the user lo identify a test operation object from a f ^^*^"^"^}^*^""i2jJlS.^,^^^^ allows 

menu of all defined and instantiated test operation objects. 55 Q^^'^^^fi'^^'^'oF'rS^ 

Element 804 is next operable to add the selected lest ^-^ilSS'^l^'SSSIw^^ a^typical 
operation object to the current playback file (alternatively, ai^Smatie^ener^^ 
the user may identify a test operation object to be deleted 
from the playback file, to be edited, or to be moved lo 

another position in the playback file). Element 806 deler- 60 ^ ~ 

L ' u . c » • . Ill BOOL CX>pe nFilc::Gcne rate () 

mines whether the user wishes to further edit the playback | 

file or whether all changes are done. If further changes are CFUeLisi *pFiicUat - nuli^ 

desired, execution continues by looping back lo element // Ask the appUcation foi a list of fties with a certain extension. 

802. If no further changes are desired, the method completes P°|l^^^' " GetAppiication( >>FindFiies(-.ui"); 

... - . , , ... // It there are none, then fait. 

With processing by element 808 to save the new playback 65 if (pFiicUst(>>Empty()) 
file to mass storage associated with the present invention. return fai^E; 

TEST OPERATION OBJECTS 
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-continued 

// Pick one of the files ranctoraly from the list. 

nPickedFile - Randoin(pFLleList( )->NumbeiOfFiles( )); 

// Oct the name of the picked file and save it, ^ 

m_strFileName = p File Lis t( )->GetFileName(nPickedFile); 

return TRUE; 

} 

BOOL COpcDFile::Manual( ) 
{ 

CFileDialog dlgFile Dialog; 10 
BOOLbRct- FALSE; 

// Bring up a dialog box asking the user for a file name. 

if (IDOK - - dlgFilcDialog.DoModalC )) 

{ 

// If the user clicks *OK' save the new file namej 

// otherwise FALSE will be returned and the edit/create j5 

// operation will be canceled. 

□i_strFileName - dlgFile Dialog.GetFileNameC ); 

bRet » TRUE; 

} 

return bRet; 



In^generalj, the Generate function is used by the testing 
toois^tb create pseu do -random test data input values. Other 
methods besides random generation may be used in a 
Generate function to automatically create appropriate test 25 
values. For example, sequencing through a variety of test 
values from low to high may be appropriate in some lest 
cases. The automatically generated values are also logged in 
the playback file so that any problems revealed by the test 
sequence with specific input values may be easily repro- 
duced. 

The exemplary Manual input function presents exemplary 
code used to display a dialog box in which a test user of the 
application may enter data input values to proceed in the test 
process. As in the Generate exemplary function, any data 
input value received is logged in the playback file so that a 
test sequence may be easily reproduced. , 

The file and memory I/O function interface method per- 
mits simulation of input data values retrieved from tempo- 
rary files or memory buffers. Such file or buffered I/O 
requests generated by invocation of the execute function of 40 
the test operation object are satisfied by simulating the 
writing and reading of values to and from files or memory 
buffers by the application program. By simulating these 
operations, the values used may be captured and thereby 
reproduced in later invocations of the playback file (i.e. to 45 
reproduce a test sequence which resulted in a failure). 

The rules interface method comprises a set of functions 
which determine the propriety of running a particular test 
operation function, or group of test operation functions in 
light of the current context of the application program, or the 50 
context of the test processing. The rules interface method is 
discussed below in additional detail. 
TEST OPERATION OBJECT GROUPS 

As a convenience to the testing user of an application 
program with the test tools embedded according to the 55 
present invention, functions are provided in the lest opera- 
lion object library (DLL) of the present invention to permit 
a*development engineer to associate related test operation 
6t?j'ects into "groups". Function calls in the DLL permit a 
development engineer to create new group definitions. Each 60 
group becomes associated with a unique group identifier. A 
^ textual name may be associated with each group for use in 
prompting a test user to enter parameters associated with the 
group. Function calls within the DLL used to define each test 
operation object include a parameter to identify the pre- 65 
defined group with which the test operation object is to be 
associated. 
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Each of the groups defined by the development engineer 
may be associated with a weighting value. Weighting values 
are used in conjunction with the pseudo-random auto -test 
mode described above to adjust the frequency distribution of 
random calls to the various test operation objects. The 
weighting value is used to determine the relative number of 
calls to test operation objects within one group as distinct 
from the number of calls to test operation objects associated 
with another group. In other words, the development engi- 
neer may define one group to be "randomly" selected more 
frequently than is another group. In addition to the group 
weighting, each test operation object within a group may be 
weighted further to determine its relative frequency of 
invocation as compared to each other test operation object in 
the same group. 

EMBEDDING TEST OBJECTS 

The test objects defined by the product development team 
must be defined and instantiated in the initialization code of 
the application program as shown in element 502 of FIG. 5. 
The following C++ code sample is exemplary of an 
approach to embedding the test object definition and instan- 
tiation in the application program. In the following exem- 
plary code, the test methods and structures of the present 
invention are referred to by the Microsoft trade name 
TestWizard. It is presumed in the exemplary code below that 
the appropriate header file(s) have been included to provide 
the required object class definitions. 



// Create the test suite object which defines the parameters 
of the pseudo- random 

// auto- test mode of operation. This object defines the groups 
of test operations 

// and the relative weights among the groups and operations. 
m__pTcstSuitc = new CTestSuitc(MAX_GROUP, MAX_OPERAnON, 

// Create the TcstWizard object - the heart of the methods and 
structure of the 
// present invention. 
m_pTestWizaid = new CTest Wizard; 

// Define five groups and their respective weights in the test suite object 

// (initially default values are used for the group weights) in the test 

suite (±iject. 

// The five groups are: 

// CREATEGROUP 

// FUJEGROUP 

// MANIPGROUP 

// LAYERGROUP 

// SELECnONGROUP 

m_pTtstSuite ->SctGroupWcight(CREATEGROUP, 
DEFAULT_WEIGHT); 

m_j)'I^stSuite->SetGioupWcight(FILEGROUP, DEFAULT_WEIGHT); 

m_pTestSuitc->SetGroupWeight(MANIPGROUP, 

DEFAULT_WEIGHT); 

m_pTfcslSuitc->SetGTOUpWcightCLAYERGROUP. 
DEFAULT_WEIGHT}; 

m_p1^stSuitc->SctGroupWcightCSELECTIONGROUP, 
DEFAULT„WEIGHT); 

m_pTestWi2ard->SetTestSuite(m_pTestSuite, ".sui", ".wiz"); 

m_4)TbstWizard->SetApplicationName("DcmoApp"); 

m^TfestWiza rd - >Se tPlaybackExtens io n(". wiz"); 

// Get access to the TbstWizard User Interface object 

m_p1festWizard->Get'IfestWi2ardUI (fiptestWizUI); 

// Define the pages to be displayed in the UI along with the 

OPTIONS page 

pteslWizU I - > AddPage(n ew CCrca te Page(p lest WizU I)); 

ptcstWizUI->AddPagc(new CFUcPage^itcstWizuI)); 

pte8tWizUI->AddPage(new CManipulalionPage(ptestWizUI)); 

ptestWizUI->AddPage(new CLayerPage(ptestWizU[)); 

ptcstWizUI->AddPage(new CSclcction(plestWizUI)); 

// Add test operation objects to the FIUE group. The test operations are: 

// NEWOP (use the CNewFtle test operation object) 

// OPENOP (Not Yet Implemented) 

// CLOSEOP (use the CCIoseRle test operation object) 
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menl and utilize the features of the present invention. Many 

-continued assumptions regarding variable types and scopes are made in 

• ' " the above code segments. One of ordinary skill in the art will 

n ^^^?c^p ?v f ^f/'^'f' r""''"" '^^"'''^ r^a^ly recognize the intended meaning of the code sample, 

// SAVEASOP (Not Yet Implemented) / ^ . , . °. r . . K I 

m_pTcstSuitc->SctOpeiBtion (FiLEGROUP, NEWOP. 5 namely: exemplary pseudo-code suggestive of the intended 

new (SKETCHNEWGROUP) use of the methods and structures of the present invention, 

CNcwFUc(m„pTcstWi2ard)); PSEUDO-RANDOM AUTO-TEST MODE USER INTER- 

m_pTestSuite->SetOperatiop(FILEGROUP, OPENOP, new FACE 

CNYiopemtion(m_pTestWizard)); FIG. 9 is an exemplary display screen used in the test tools 

m„pTestSuite->SetOperation(FILEGROUP, CLOSEOP, j r *u * • »• . 

new (SKETCHNEWGROUP) 10 methods ot the present invention to permit the test user 

caoscFiIc(m_pTcstWizard)); to define the weight values used for each test operation 

m_pTestSuiie->SetOperation(FiLEGROUP, SAVEOP, object and for cach group of tcst Operation objects. The set 

new (SKETCHNEWGROUP)^ of test operation objects and their groupings are defined by 

m_p?«tSuii^S7i^?ra^^ SAVEASOP, new P^^^^^^ development group when the objects are embed- 

CNYIopcration(m_pTcstWizard)); ded in the Source code of the application program (as 

// Invoke the UI to display the user interface and begin testing. discUSSed above with respect tO FIGS. 2 and 4. Label 900 of 

m_pTe3tWizard->lnvokeUl( ); pjG. 9^ discussed in detail below with respect to FIG. 10, 

indicates the hidden "OPTIONS" display screen used to set 
The above, exemplary initialization code may be inserted global option values for the pseudo-random auto-test mode 
in the application program at any desirable position in the 20 ^P^^^^o"- fo^^ exemplary groups have been 
initialization of the application. The invocation of the defined as indicated by the labels 902-910. Specifically the 
testWizard User Interface (to commence testing operations) groups are named: "CREATION/DELETION" (label 902), 
may be inserted in the application at any desirable position "FILE" (label 904). "MANIPULATION" (label 906), 
including: at startup of the application as shown above. "LAYER" (label 908), AND "SELECTION" (label 910). 
following operation of some particular function, as an option 25 display screen of FIG. 9 is an exemplary screen for 
in the standard user interface of the application (i.e. in a ^^try of weighting values by the test user to be associated 
menu item of the application program), or at any other with the test group named "CREATION/DELETION." The 
desirable point in the operation of the program. group weight value entered in field 912 sets the group weight 
The SetOperation function calls specify the test operation fo^* this test operation group relative to all other test opera- 
object (third parameter) to be associated with the group and 30 tion groups defined by the product development group. For 
operation specified by the first two parameters. The test example, if the other four groups (labeled "FILE", 
operation objects are derived from a COPERAHON test "MANIPULATION", "LAYER", and "SELECTION") have 
operation object class defined by the methods and structures group weights of 75, 25, 100, and 125, respectively, then the 
of the present invention (and included by appropriate header test operations in this test group ("CREATION/ 
files in the exemplary code above). The CNewFilc, 35 DELETION") will be selected (50/(50+75+25+100+125)) 
CCloseFile, and CSaveFile tcst operation objects are derived times or 13.33% of the pseudo-random selections, 
from the COperation object class. The CNYIOperation Within each test group, each lest operation may be indi- 
subclass of the COPERAnON class is defined by the vidually weighted and enabled/disabled. In the 
methods and structures of the present invention to provide a "CREATION/DELETION" test group of FIG. 9, there are 
default test operation object as a place holder for functions 40 six test operations labeled: "CREATE", "DUPLICATE", 
not yet implemented by the development group. "PASTE", "CUT", "COPY", and "DELETE." These six test 
A COPERATION class object may be derived from the operations may be individually enabled or disabled by 
base class in accord with the following C++ code sample: checking or clearing the associated check box, 954-964, 
CNewFile::CNewFile(CTestWizard*pTestWizard) respectively. In addition, each of these six test operation may 
:COperation(pTestWizard) 45 individually weighted by entermg a weight value in the 
r associated weight field, 814-924, respectively. The enabled 
^ test operations within the group will be pseudo-randomly 
//Body of code to implement a new file operation selected according to the ratio of their respective weights 
} whenever their containing group, "CREATION/ 
The execute interface method function must be imple- DELETION", is selected. In other words, the "CREATE" 
mented in every derived object. A skeletal exemplary lest operation will be selected 50/(50+10+25) times, or 
execute function might appear as follows: 58.82% of the limes when the "CREATION/DELETION" 



DWORD CNewFiIe::Execute( ) 
{ 

CString str Message; 

str Message. Forinat(" File New Called\n"); 
if(Log(str Message)) 

AfxGetMainWnd( )->SendMessagc(WM_COMMAND, ID_FILE_NEW); 
return Error( ); 

} 



One of ordinary skill in the art will recognize that the ^5 
above exemplary code segments are intended only as group is selected. Since the weighting of the group is 
demonstrative of the structure of actual code used to imple- multiplied by the test operation weight, the "CREATE" test 
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Operation will be selected 7.84% of the lime among all 
defined and enabled test operation objects. 

Button fields 926-932 are used to save and retrieve the 
values and settings entered in the above discussed fields 
(912-924 and 954-964) for future reference. Use of buttons 
such as these are well known to those of ordinary skill in the 
art and need not be discussed further. 

Button field 934 begins the pseudo-random auto-test 
mode operations as defined by the parameters entered in the 
above discussed fields (912-924 and 954-964). FIGS. 5-^8, 
discussed above, present the detailed operation of the 
pseudo-random auto-test mode. 

FIG. 10 is an exemplary display screen used to prompt the 
test user of the present invention to enter parameters 
(options) used in the pseudo-random auto-test mode of 
operation. Label 900 of HG. 10 indicates the semantic use 
of the screen display for entry of "OPTIONS*' in the running 
of the pseudo-random auto-test mode. Labels 902-910, as 
discussed above with reference to FIG. 9, indicate the names 
of exemplary groups of lest operation objects as defined by 
the development. 

Fields on the "OPTIONS" screen of FIG. 10 are used to 
define parameters which control the operation of the pseudo- 
random auio-tesl mode of the present invention. Field 1012 
is used to enter the overall error or failure threshold value. 
This value determines the maximum number of erroneous 
results permitted of all lest operation invocations before the 
pseudo-random auto-test mode is terminated. Use of this 
overall threshold value is discussed above with respect to 
element 608 of FIG. 6A and elements 622 and 624 of FIG. 
6B. Field 1014 of FIG. 10 is used to enter a single failure 
threshold value for use in the methods of the present 
invention as discussed above with reference to RGS. 5-8. 
The single failure threshold value is used to determine 
whether any particular test operation has exceeded this 
failure threshold. The lest tools and methods of the present 
invention account for the number of test operation failures 
associated with the invocation of each test operation object 
as well as the total number of test failures. This threshold 
value is used in a manner similar to that of the overall test 
failure threshold to terminate the invocation of a particular 
lest operation object in the pseudo-random auto-test mode of 
operation. As shown in FIG. 10, threshold values of zero (or 
less) will terminate the pseudo-random auto test mode (field 
1012) or the continued invocation of a particular test (field 
1014) as soon as any failure occurs in the invocation of a lest 
operation object. 

Field 1016 is used to enter the random number generator 
seed value for the start of the pseudo -random test selection. 
To reproduce a particular "random" test sequence, the user 
may enter the same seed value as used in a previous run of 
that test sequence. Field 1018 is used to enter the total 
number of test operation objects to be invoked for the 
pseudo-random test sequence. The pseudo-random auto-test 
mode will terminate operation after this number of test 
operation objects are invoked (regardless of the success or 
failure of the test operation). Field 1020 specifies the direc- 
tory on a mass storage device associated with the computing 
environment running the test. Files used in the operation of 
the pseudo-random auto-test mode (playback and result log 
files, etc.) will be stored in this directory for later reference. 

Check box field 1022 is used to enable the persistent 
storage of single operation threshold values and error counts 
from previous invocations of test operations to accumulate 
the error counters. When field 1022 is checked, test opera- 
tions which previously produced erroneous results exceed- 
ing the designated threshold value are skipped. Otherwise, 
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all test operations are invoked normally. This feature is 
useful to proceed beyond known problems in a test sequence 
to reveal other new problems. This "persistent" error infor- 
mation is retained and accumulated through all lest opera- 
lion invocations and through a plurality of test sequence 
runs. Field 1024 is a button which, when selected 
("clicked") by the user, will clear the persistent log of 
erroneous results to again permit testing of all operations in 
a lest sequence of the pseudo-random auto-iesi mode. 

Field 1026 is used to enter a maximum group/operation 
weight value. This maximum value is used to automatically 
generate weighted values for all groups and for all opera- 
tions within a group when button 1028 is clicked. 

Button fields 1030-1036 are used to save and retrieve the 
values and settings entered in the above discussed fields 
(1012-1028) for future reference. Use of buttons such as 
these are well known to those of ordinary skill in the art and 
need not be discussed further. 

Button field 1038 begins the pseudo-random auto-tesl 
mode operations as defined by the parameters entered in the 
above discussed fields (1012-1028). FIGS. 5-8, discussed 
above, present the detailed operation of the pseudo-random 
auto-test mode. 

PLAYBACK FILE EDITING USER INTERFACE 

FIG. 11 is an exemplary display screen designed to permit 
a test user to edit the contents of a previously recorded 
playback file. In the playback file creation mode of 
operation, a lest user edit a previously recorded playback file 
to customize the lest sequence, or create a new test sequence. 
In the screen display of FIG. U, a list of available lest 
operations 1100 is positioned on the left side of the screen. 
A list of the lest operations currently selected for the lest 
sequence, the current script 1102, appears on the right side 
of the screen. A user may highlight ("click" on) a desired lest 
operation from the lisl of available lest operations, then click 
on the "ADD" button 1104 to add the highlighted test 
operation to the current playback 1102. Conversely, the user 
may highlight a test operation in the current script 1102 and 
click the "REMOVE" button 1106 to remove the highlighted 
test operation from the current script 1102. 

In addition to the "ADD" and "REMOVE" functions, a 
highlighted test operation in the current playback 1102 may 
be skipped by clicking the "DISABLE" button 1108. A 
pause in the lest sequence may be added to the current script 
by highlighting a test operation in the current script 1102 and 
clicking the "BREAK" button 1110. This marks the high- 
lighted lest operation such that the operation may pause 
before executing. 

Any test operation in the current playback 1102 may be 
altered by highlighting the lest operation and clicking on the 
"EDIT" button 1112. 

Test operations in the current list 1102 may be moved up 
or down in the current lisl 1102 relative to other lest 
operations by highlighting the test operation to be moved 
and then clicking on the "MOVE UP' button 1114 or 
"MOVE DOWN" button 1116. 

Button fields 1118-1126 are used to save and retrieve the 
values and settings entered in the above discussed fields 
(1100-1102) for future reference. Use of buttons such as 
these are well known to those of ordinary skill in the art and 
need not be discussed further. 
RULES 

As shown in FIG. 6B, element 612 is operable to adjust 
the weight selection tables used in pseudo- random auto-test 
mode for any changed weights. Rules as used herein, are 
functions which are invoked in association with the invo- 
cation of the execute interface function wherein the rules 
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. function modify the current weight value for one or more 
test operation objects or groups. By modifying the weight 
value of a particular test operation object or a particular 
group, the rules function may alter the frequency with which 
certain test execute interface functions are invoked. The 5 
conditions in which the rules function will modify a weight 
value for a selected test operation object or group, may be 
completely defined by the code implementing the rules 
function as written by the product development group. 

Rules, as defined herein, may be used for example, to jq 
reduce the frequency of a particular test operation depending 
on the current state of operation of the application program, 
or for example, depending upon the current state of progress 
in running the test application program. Changing the fre- 
quency of invocation of a particular test operation object or 
group may include, for example, reducing its frequency of 
invocation to zero (i.e. "not available**). For example, func- 
tions which perform an edit menu copy or paste function 
may be valid only when a file is "open** in the context of the 
application program. Rule functions may be used to assure 20 
that the execute interface functions for the copy or paste test 
operation objects will never be called until after the test 
operation object execute interface function for the files/open 
menu operation is invoked successfully. 

The rules interface functions for each test operation object 25 
are invoked to adjust the weighted selection table before the 
next random selection is made in the flowcharts of FIGS. 6A 
and 6B. 

What is claimed is: 

1. In a computer application program operable on a data 3Q 
processing system, a method, integrated with said computer 
application program, for testing said computer application 
program comprising the steps of: 

(a) embedding at least one test operation object definition 
within a program structure of said computer application 35 
program; 

(b) calling a function to instantiate at least one test 
operation object from said at least one test operation 
object definition, wherein said test operation object 
defines a function of said computer application pro- 40 
gram to be tested, and wherein said test operation 
objects includes an execute interface method; and 

(c) calling said execute interface method defined by said 
lest operation object to generate a test result. 

2. The method of claim 1 further comprising the step of 45 
recording said test result. 

3. The method of claim 1 wherein step (b) instantiates a 
plurality of test operation objects from a plurality of embed- 
ded test operation object definitions each said plurality of 
test operation objects defining a function of said computer 50 
application program to be tested, and wherein said test 
operation objects each include an execute interface method. 

4. The method of claim 3 wherein step (c) includes the 
step of selecting one of said plurality of test operation 
objects. 55 

5. The method of claim 4 further comprising the step of 
repeating steps (b) and (c). 

6. The method of claim 5 further comprising the step of 
recording the selected one of said plurality of test operations. 

7. The method of claim 6 wherein said step of selecting 60 
comprises the step of picking said one of said plurality of 
test operation objects from among a plurality of test opera- 
tion objects previously recorded. 

8. The method of claim 4 wherein said step of selecting 
comprises the step of pseudo-randomly picking said one of 65 
said plurality of test operation objects from among said 
plurality of lest operation objects. 



9. The method of claim 4 wherein each of said plurality 
of test operation objects further includes rule elements 
defining when said each of said plurality of test operation 
objects may be executed relative to others of said plurality 
of test operation objects. 

10. The method of claim 9 wherein said step of selecting 
further comprises the step of applying said rule elements to 
determine that a lest operation object may be selected. 

11. In a computer application program operable on a data 
processing system, a method for integrating a computer 
application program with the testing thereof comprising the 
steps of: 

(a) embedding test operation object definitions in said 
computer application program; 

(b) instantiating at least one test operation object defined 
by operation of step (a), wherein said test operation 
object defines a function of said computer application 
program to be tested, and wherein said test operation 
objects includes an execute interface method; 

(c) calling said execute interface method included in said 
test operation object to generate a test result; and 

(d) recording said test result for pass-fail analysis of said 
computer application program. 

12. The method of claim 11 wherein step (b) instantiates 
a plurality of test operation objects each defining a function 
of said computer application program to be tested, and 
wherein said lest operation objects each include an execute 
interface method. 

13. The method of claim 12 wherein step (b) includes the 
step of selecting one of said plurality of test operation 
objects. 

14. The method of claim 13 further comprising the step of 
repeating steps (b), (c) and (d). 

15. The method of claim 14 further comprising the step of 
recording the selected one of said plurality of test operations. 

16. The method of claim 15 wherein said step of selecting 
comprises the step of picking said one of said plurality of 
test operation objects from among a plurality of test opera- 
tion objects previously selected and recorded. 

17. The method of claim 13 wherein said step of selecting 
comprises the step of pseudo-randomly picking said one of 
said plurality of test operation objects from among said 
plurality of test operation objects. 

18. The method of claim 13 wherein each of said plurality 
of test operation objects further includes rule elements 
defining when said each of said plurality of test operation 
objects may be executed relafive to others of said plurality 
of test operation objects. 

19. The method of claim 18 wherein said step of selecting 
further comprises the step of applying said rule elements to 
determine that a lest operation object may be selected, 

20. A computer readable storage device, readable by a 
computer, embodying a program of computer instructions 
executable by a computer to perform the method steps for 
integrating testing of the program with execution of the 
program, the method comprising the steps of: 

(a) embedding at least one test operation object definition 
within a program structure of said computer application 
program; 

(b) caUing a function to instantiate at least one test 
operation object from said at least one test operation 
object definition, wherein said lest operation object 
defines a function of said computer application pro- 
gram to be tested, and wherein said test operation 
objects includes an execute interface method; and 

(c) calling said execute interface method defined by said 
test operation object to generate a test result. 
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21. The storage device of claim 20 wherein the method 
further comprises the step of recording said test result. 

22. The storage device of claim 20 wherein the method 
step (b) instantiates a plurality of test operation objects from 
a plurality of embedded test operation object definitions 
each said plurality of test operation objects defining a 
function of said computer application program to be tested, 
and wherein said test operation objects each include an 
execute interface method. 

23. The storage device of claim 22 wherein the method 
step (c) includes the step of selecting one of said plurality of 
test operation objects. 

24. The storage device of claim 23 wherein the method 
further comprises the step of repeating steps (b) and (c). 

25. The storage device of claim 24 wherein the method 
further comprises the step of recording the selected one of 
said plurality of lest operations. 

26. The storage device of claim 25 wherein the method 
step of selecting comprises the step of picking said one of 
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said plurality of test operation objects from among a plu- 
rality of test operation objects previously recorded. 

27. The storage device of claim 23 wherein the method 
^ step of selecting comprises the step of pseudo-randomly 

picking said one of said plurality of test operation objects 
from among said plurality of test operation objects. 

28. The storage device of claim 23 wherein each of said 
plurahty of test operation objects further includes rule 

10 elements defining when said each of said plurality of test 
operation objects may be executed relative to others of said 
plurality of test operation objects. 

29. The storage device of claim 28 wherein the method 
15 step of selecting further comprises the step of applying said 

rule elements to determine that a test operation object may 
be selected. 



09/09/2004, EAST Version: 1.4.1 



